3d object annotation

ABSTRACT

Disclosed herein are systems and methods for presenting and annotating virtual content. According to an example method, a virtual object is presented to a first user at a first position via a transmissive display of a wearable device. A first input is received from the first user. In response to receiving the first input, a virtual annotation is presented at a first displacement from the first position. A first data is transmitted to a second user, the first data associated with the virtual annotation and the first displacement. A second input is received from the second user. In response to receiving the second input, the virtual annotation is presented to the first user at a second displacement from the first position. Second data is transmitted to a remote server, the second data associated with the virtual object, the virtual annotation, the second displacement, and the first position.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. Non-Provisional applicationSer. No. 17/174,141, filed Feb. 11, 2021, which claims benefit of U.S.Provisional Application No. 62/977,073, filed Feb. 14, 2020, thecontents of which is incorporated herein by reference in its entirety.

FIELD

This disclosure relates in general to systems and methods for presentingand annotating virtual content, and in particular to systems and methodsfor presenting and annotating virtual content in a mixed realityenvironment.

BACKGROUND

Virtual environments are ubiquitous in computing environments, findinguse in video games (in which a virtual environment may represent a gameworld); maps (in which a virtual environment may represent terrain to benavigated); simulations (in which a virtual environment may simulate areal environment); digital storytelling (in which virtual characters mayinteract with each other in a virtual environment); and many otherapplications. Modern computer users are generally comfortableperceiving, and interacting with, virtual environments. However, users'experiences with virtual environments can be limited by the technologyfor presenting virtual environments. For example, conventional displays(e.g., 2D display screens) and audio systems (e.g., fixed speakers) maybe unable to realize a virtual environment in ways that create acompelling, realistic, and immersive experience.

Virtual reality (“VR”), augmented reality (“AR”), mixed reality (“MR”),and related technologies (collectively, “XR”) share an ability topresent, to a user of an XR system, sensory information corresponding toa virtual environment represented by data in a computer system. Thisdisclosure contemplates a distinction between VR, AR, and MR systems(although some systems may be categorized as VR in one aspect (e.g., avisual aspect), and simultaneously categorized as AR or MR in anotheraspect (e.g., an audio aspect)). As used herein, VR systems present avirtual environment that replaces a user's real environment in at leastone aspect; for example, a VR system could present the user with a viewof the virtual environment while simultaneously obscuring his or herview of the real environment, such as with a light-blocking head-mounteddisplay. Similarly, a VR system could present the user with audiocorresponding to the virtual environment, while simultaneously blocking(attenuating) audio from the real environment.

VR systems may experience various drawbacks that result from replacing auser's real environment with a virtual environment. One drawback is afeeling of motion sickness that can arise when a user's field of view ina virtual environment no longer corresponds to the state of his or herinner ear, which detects one's balance and orientation in the realenvironment (not a virtual environment). Similarly, users may experiencedisorientation in VR environments where their own bodies and limbs(views of which users rely on to feel “grounded” in the realenvironment) are not directly visible. Another drawback is thecomputational burden (e.g., storage, processing power) placed on VRsystems which must present a full 3D virtual environment, particularlyin real-time applications that seek to immerse the user in the virtualenvironment. Similarly, such environments may need to reach a very highstandard of realism to be considered immersive, as users tend to besensitive to even minor imperfections in virtual environments—any ofwhich can destroy a user's sense of immersion in the virtualenvironment. Further, another drawback of VR systems is that suchapplications of systems cannot take advantage of the wide range ofsensory data in the real environment, such as the various sights andsounds that one experiences in the real world. A related drawback isthat VR systems may struggle to create shared environments in whichmultiple users can interact, as users that share a physical space in thereal environment may not be able to directly see or interact with eachother in a virtual environment.

As used herein, AR systems present a virtual environment that overlapsor overlays the real environment in at least one aspect. For example, anAR system could present the user with a view of a virtual environmentoverlaid on the user's view of the real environment, such as with atransmissive head-mounted display that presents a displayed image whileallowing light to pass through the display into the user's eye.Similarly, an AR system could present the user with audio correspondingto the virtual environment, while simultaneously mixing in audio fromthe real environment. Similarly, as used herein, MR systems present avirtual environment that overlaps or overlays the real environment in atleast one aspect, as do AR systems, and may additionally allow that avirtual environment in an MR system may interact with the realenvironment in at least one aspect. For example, a virtual character ina virtual environment may toggle a light switch in the real environment,causing a corresponding light bulb in the real environment to turn on oroff. As another example, the virtual character may react (such as with afacial expression) to audio signals in the real environment. Bymaintaining presentation of the real environment, AR and MR systems mayavoid some of the aforementioned drawbacks of VR systems; for instance,motion sickness in users is reduced because visual cues from the realenvironment (including users' own bodies) can remain visible, and suchsystems need not present a user with a fully realized 3D environment inorder to be immersive. Further, AR and MR systems can take advantage ofreal world sensory input (e.g., views and sounds of scenery, objects,and other users) to create new applications that augment that input.

XR systems may be uniquely positioned to enable greater collaborationbetween people. The ability to present virtual content in a persistentand three-dimensional manner can allow people to more naturally interactwith virtual content. For example, arranging virtual objects inthree-dimensional space may enable much more natural location recallthan what two-dimensional screens can offer. Where a user of atwo-dimensional screen may have to hunt through one of forty open tabsto re-open a desired application, a user of an XR system may be able topinpoint a desired virtual object displayed on a desk (like picking up areal folder placed on a desk). Furthermore, XR systems may enable usersto see virtual avatars of other users to simulate the live presence ofother people. This may enable more natural collaboration than what atelephone call or even videoconference can offer. It can therefore bedesirable to develop systems and methods for enabling deep usercollaboration on XR systems.

XR systems can offer a uniquely heightened sense of immersion andrealism by combining virtual visual and audio cues with real sights andsounds. Accordingly, it is desirable in some XR systems to present avirtual environment that enhances, improves, or alters a correspondingreal environment. This disclosure relates to XR systems that enableconsistent placement of virtual objects across multiple XR systems.

BRIEF SUMMARY

Examples of the disclosure describe systems and methods for presentingand annotating virtual content. According to an example method, avirtual object is presented to a first user at a first position via atransmissive display of a wearable device. A first input is receivedfrom the first user. In response to receiving the first input, a virtualannotation is presented at a first displacement from the first positionvia the transmissive display. A first data is transmitted to a seconduser, the first data associated with the virtual annotation and thefirst displacement. A second input is received from the second user. Inresponse to receiving the second input, the virtual annotation ispresented to the first user at a second displacement from the firstposition via the transmissive display. Second data is transmitted to aremote server, the second data associated with the virtual object, thevirtual annotation, the second displacement, and the first position.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1C illustrate an example mixed reality environment, accordingto some embodiments.

FIGS. 2A-2D illustrate components of an example mixed reality systemthat can be used to generate and interact with a mixed realityenvironment, according to some embodiments.

FIG. 3A illustrates an example mixed reality handheld controller thatcan be used to provide input to a mixed reality environment, accordingto some embodiments.

FIG. 3B illustrates an example auxiliary unit that can be used with anexample mixed reality system, according to some embodiments.

FIG. 4 illustrates an example functional block diagram for an examplemixed reality system, according to some embodiments.

FIGS. 5A-5C illustrate an example of a mixed reality collaborationsession, according to some embodiments.

FIG. 6 illustrates an example of a session manager architecture,according to some embodiments.

FIG. 7 illustrates an example of a session instance, according to someembodiments.

FIG. 8 illustrates an example of a mixed reality collaboration session,according to some embodiments.

FIG. 9 illustrates an example of an annotation menu, according to someembodiments.

FIG. 10 illustrates an example of mixed reality annotation, according tosome embodiments.

FIG. 11 illustrates an example of mixed reality annotation, according tosome embodiments.

DETAILED DESCRIPTION

In the following description of examples, reference is made to theaccompanying drawings which form a part hereof, and in which it is shownby way of illustration specific examples that can be practiced. It is tobe understood that other examples can be used and structural changes canbe made without departing from the scope of the disclosed examples.

Mixed Reality Environment

Like all people, a user of a mixed reality system exists in a realenvironment that is, a three-dimensional portion of the “real world,”and all of its contents, that are perceptible by the user. For example,a user perceives a real environment using one's ordinary human sensessight, sound, touch, taste, smell—and interacts with the realenvironment by moving one's own body in the real environment. Locationsin a real environment can be described as coordinates in a coordinatespace; for example, a coordinate can include latitude, longitude, andelevation with respect to sea level; distances in three orthogonaldimensions from a reference point; or other suitable values. Likewise, avector can describe a quantity having a direction and a magnitude in thecoordinate space.

A computing device can maintain, for example in a memory associated withthe device, a representation of a virtual environment. As used herein, avirtual environment is a computational representation of athree-dimensional space. A virtual environment can includerepresentations of any object, action, signal, parameter, coordinate,vector, or other characteristic associated with that space. In someexamples, circuitry (e.g., a processor) of a computing device canmaintain and update a state of a virtual environment; that is, aprocessor can determine at a first time t0, based on data associatedwith the virtual environment and/or input provided by a user, a state ofthe virtual environment at a second time t1. For instance, if an objectin the virtual environment is located at a first coordinate at time t0,and has certain programmed physical parameters (e.g., mass, coefficientof friction); and an input received from user indicates that a forceshould be applied to the object in a direction vector; the processor canapply laws of kinematics to determine a location of the object at timet1 using basic mechanics. The processor can use any suitable informationknown about the virtual environment, and/or any suitable input, todetermine a state of the virtual environment at a time t1. Inmaintaining and updating a state of a virtual environment, the processorcan execute any suitable software, including software relating to thecreation and deletion of virtual objects in the virtual environment;software (e.g., scripts) for defining behavior of virtual objects orcharacters in the virtual environment; software for defining thebehavior of signals (e.g., audio signals) in the virtual environment;software for creating and updating parameters associated with thevirtual environment; software for generating audio signals in thevirtual environment; software for handling input and output; softwarefor implementing network operations; software for applying asset data(e.g., animation data to move a virtual object over time); or many otherpossibilities.

Output devices, such as a display or a speaker, can present any or allaspects of a virtual environment to a user. For example, a virtualenvironment may include virtual objects (which may includerepresentations of inanimate objects; people; animals; lights; etc.)that may be presented to a user. A processor can determine a view of thevirtual environment (for example, corresponding to a “camera” with anorigin coordinate, a view axis, and a frustum); and render, to adisplay, a viewable scene of the virtual environment corresponding tothat view. Any suitable rendering technology may be used for thispurpose. In some examples, the viewable scene may include only somevirtual objects in the virtual environment, and exclude certain othervirtual objects. Similarly, a virtual environment may include audioaspects that may be presented to a user as one or more audio signals.For instance, a virtual object in the virtual environment may generate asound originating from a location coordinate of the object (e.g., avirtual character may speak or cause a sound effect); or the virtualenvironment may be associated with musical cues or ambient sounds thatmay or may not be associated with a particular location. A processor candetermine an audio signal corresponding to a “listener” coordinate—forinstance, an audio signal corresponding to a composite of sounds in thevirtual environment, and mixed and processed to simulate an audio signalthat would be heard by a listener at the listener coordinate—and presentthe audio signal to a user via one or more speakers.

Because a virtual environment exists only as a computational structure,a user cannot directly perceive a virtual environment using one'sordinary senses. Instead, a user can perceive a virtual environment onlyindirectly, as presented to the user, for example by a display,speakers, haptic output devices, etc. Similarly, a user cannot directlytouch, manipulate, or otherwise interact with a virtual environment; butcan provide input data, via input devices or sensors, to a processorthat can use the device or sensor data to update the virtualenvironment. For example, a camera sensor can provide optical dataindicating that a user is trying to move an object in a virtualenvironment, and a processor can use that data to cause the object torespond accordingly in the virtual environment.

A mixed reality system can present to the user, for example using atransmissive display and/or one or more speakers (which may, forexample, be incorporated into a wearable head device), a mixed realityenvironment (“MRE”) that combines aspects of a real environment and avirtual environment. In some embodiments, the one or more speakers maybe external to the head-mounted wearable unit. As used herein, an MRE isa simultaneous representation of a real environment and a correspondingvirtual environment. In some examples, the corresponding real andvirtual environments share a single coordinate space; in some examples,a real coordinate space and a corresponding virtual coordinate space arerelated to each other by a transformation matrix (or other suitablerepresentation). Accordingly, a single coordinate (along with, in someexamples, a transformation matrix) can define a first location in thereal environment, and also a second, corresponding, location in thevirtual environment; and vice versa.

In an MRE, a virtual object (e.g., in a virtual environment associatedwith the MRE) can correspond to a real object (e.g., in a realenvironment associated with the MRE). For instance, if the realenvironment of an MRE includes a real lamp post (a real object) at alocation coordinate, the virtual environment of the MRE may include avirtual lamp post (a virtual object) at a corresponding locationcoordinate. As used herein, the real object in combination with itscorresponding virtual object together constitute a “mixed realityobject.” It is not necessary for a virtual object to perfectly match oralign with a corresponding real object. In some examples, a virtualobject can be a simplified version of a corresponding real object. Forinstance, if a real environment includes a real lamp post, acorresponding virtual object may include a cylinder of roughly the sameheight and radius as the real lamp post (reflecting that lamp posts maybe roughly cylindrical in shape). Simplifying virtual objects in thismanner can allow computational efficiencies, and can simplifycalculations to be performed on such virtual objects. Further, in someexamples of an MRE, not all real objects in a real environment may beassociated with a corresponding virtual object. Likewise, in someexamples of an MRE, not all virtual objects in a virtual environment maybe associated with a corresponding real object. That is, some virtualobjects may solely in a virtual environment of an MRE, without anyreal-world counterpart.

In some examples, virtual objects may have characteristics that differ,sometimes drastically, from those of corresponding real objects. Forinstance, while a real environment in an MRE may include a green,two-armed cactus—a prickly inanimate object—a corresponding virtualobject in the MRE may have the characteristics of a green, two-armedvirtual character with human facial features and a surly demeanor. Inthis example, the virtual object resembles its corresponding real objectin certain characteristics (color, number of arms); but differs from thereal object in other characteristics (facial features, personality). Inthis way, virtual objects have the potential to represent real objectsin a creative, abstract, exaggerated, or fanciful manner; or to impartbehaviors (e.g., human personalities) to otherwise inanimate realobjects. In some examples, virtual objects may be purely fancifulcreations with no real-world counterpart (e.g., a virtual monster in avirtual environment, perhaps at a location corresponding to an emptyspace in a real environment).

Compared to VR systems, which present the user with a virtualenvironment while obscuring the real environment, a mixed reality systempresenting an MRE affords the advantage that the real environmentremains perceptible while the virtual environment is presented.Accordingly, the user of the mixed reality system is able to use visualand audio cues associated with the real environment to experience andinteract with the corresponding virtual environment. As an example,while a user of VR systems may struggle to perceive or interact with avirtual object displayed in a virtual environment—because, as notedabove, a user cannot directly perceive or interact with a virtualenvironment—a user of an MR system may find it intuitive and natural tointeract with a virtual object by seeing, hearing, and touching acorresponding real object in his or her own real environment. This levelof interactivity can heighten a user's feelings of immersion,connection, and engagement with a virtual environment. Similarly, bysimultaneously presenting a real environment and a virtual environment,mixed reality systems can reduce negative psychological feelings (e.g.,cognitive dissonance) and negative physical feelings (e.g., motionsickness) associated with VR systems. Mixed reality systems furtheroffer many possibilities for applications that may augment or alter ourexperiences of the real world.

FIG. 1A illustrates an example real environment 100 in which a user 110uses a mixed reality system 112. Mixed reality system 112 may include adisplay (e.g., a transmissive display) and one or more speakers, and oneor more sensors (e.g., a camera), for example as described below. Thereal environment 100 shown includes a rectangular room 104A, in whichuser 110 is standing; and real objects 122A (a lamp), 124A (a table),126A (a sofa), and 128A (a painting). Room 104A further includes alocation coordinate 106, which may be considered an origin of the realenvironment 100. As shown in FIG. 1A, an environment/world coordinatesystem 108 (comprising an x-axis 108X, a y-axis 108Y, and a z-axis 108Z)with its origin at point 106 (a world coordinate), can define acoordinate space for real environment 100. In some embodiments, theorigin point 106 of the environment/world coordinate system 108 maycorrespond to where the mixed reality system 112 was powered on. In someembodiments, the origin point 106 of the environment/world coordinatesystem 108 may be reset during operation. In some examples, user 110 maybe considered a real object in real environment 100; similarly, user110's body parts (e.g., hands, feet) may be considered real objects inreal environment 100. In some examples, a user/listener/head coordinatesystem 114 (comprising an x-axis 114X, a y-axis 114Y, and a z-axis 114Z)with its origin at point 115 (e.g., user/listener/head coordinate) candefine a coordinate space for the user/listener/head on which the mixedreality system 112 is located. The origin point 115 of theuser/listener/head coordinate system 114 may be defined relative to oneor more components of the mixed reality system 112. For example, theorigin point 115 of the user/listener/head coordinate system 114 may bedefined relative to the display of the mixed reality system 112 such asduring initial calibration of the mixed reality system 112. A matrix(which may include a translation matrix and a Quaternion matrix or otherrotation matrix), or other suitable representation can characterize atransformation between the user/listener/head coordinate system 114space and the environment/world coordinate system 108 space. In someembodiments, a left ear coordinate 116 and a right ear coordinate 117may be defined relative to the origin point 115 of theuser/listener/head coordinate system 114. A matrix (which may include atranslation matrix and a Quaternion matrix or other rotation matrix), orother suitable representation can characterize a transformation betweenthe left ear coordinate 116 and the right ear coordinate 117, anduser/listener/head coordinate system 114 space. The user/listener/headcoordinate system 114 can simplify the representation of locationsrelative to the user's head, or to a head-mounted device, for example,relative to the environment/world coordinate system 108. UsingSimultaneous Localization and Mapping (SLAM), visual odometry, or othertechniques, a transformation between user coordinate system 114 andenvironment coordinate system 108 can be determined and updated inreal-time.

FIG. 1B illustrates an example virtual environment 130 that correspondsto real environment 100. The virtual environment 130 shown includes avirtual rectangular room 104B corresponding to real rectangular room104A; a virtual object 122B corresponding to real object 122A; a virtualobject 124B corresponding to real object 124A; and a virtual object 126Bcorresponding to real object 126A. Metadata associated with the virtualobjects 122B, 124B, 126B can include information derived from thecorresponding real objects 122A, 124A, 126A. Virtual environment 130additionally includes a virtual monster 132, which does not correspondto any real object in real environment 100. Real object 128A in realenvironment 100 does not correspond to any virtual object in virtualenvironment 130. A persistent coordinate system 133 (comprising anx-axis 133X, a y-axis 133Y, and a z-axis 133Z) with its origin at point134 (persistent coordinate), can define a coordinate space for virtualcontent. The origin point 134 of the persistent coordinate system 133may be defined relative/with respect to one or more real objects, suchas the real object 126A. A matrix (which may include a translationmatrix and a Quaternion matrix or other rotation matrix), or othersuitable representation can characterize a transformation between thepersistent coordinate system 133 space and the environment/worldcoordinate system 108 space. In some embodiments, each of the virtualobjects 122B, 124B, 126B, and 132 may have their own persistentcoordinate point relative to the origin point 134 of the persistentcoordinate system 133. In some embodiments, there may be multiplepersistent coordinate systems and each of the virtual objects 122B,124B, 126B, and 132 may have their own persistent coordinate pointrelative to one or more persistent coordinate systems.

Persistent coordinate data may be coordinate data that persists relativeto a physical environment. Persistent coordinate data may be used by MRsystems (e.g., MR system 112, 200) to place persistent virtual content,which may not be tied to movement of a display on which the virtualobject is being displayed. For example, a two-dimensional screen mayonly display virtual objects relative to a position on the screen. Asthe two-dimensional screen moves, the virtual content may move with thescreen. In some embodiments, persistent virtual content may be displayedin a corner of a room. An MR user may look at the corner, see thevirtual content, look away from the corner (where the virtual contentmay no longer be visible because the virtual content may have moved fromwithin the user's field of view to a location outside the user's fieldof view due to motion of the user's head), and look back to see thevirtual content in the corner (similar to how a real object may behave).

In some embodiments, persistent coordinate data (e.g., a persistentcoordinate system and/or a persistent coordinate frame) can include anorigin point and three axes. For example, a persistent coordinate systemmay be assigned to a center of a room by an MR system. In someembodiments, a user may move around the room, out of the room, re-enterthe room, etc., and the persistent coordinate system may remain at thecenter of the room (e.g., because it persists relative to the physicalenvironment). In some embodiments, a virtual object may be displayedusing a transform to persistent coordinate data, which may enabledisplaying persistent virtual content. In some embodiments, an MR systemmay use simultaneous localization and mapping to generate persistentcoordinate data (e.g., the MR system may assign a persistent coordinatesystem to a point in space). In some embodiments, an MR system may mapan environment by generating persistent coordinate data at regularintervals (e.g., an MR system may assign persistent coordinate systemsin a grid where persistent coordinate systems may be at least withinfive feet of another persistent coordinate system).

In some embodiments, persistent coordinate data may be generated by anMR system and transmitted to a remote server. In some embodiments, aremote server may be configured to receive persistent coordinate data.In some embodiments, a remote server may be configured to synchronizepersistent coordinate data from multiple observation instances. Forexample, multiple MR systems may map the same room with persistentcoordinate data and transmit that data to a remote server. In someembodiments, the remote server may use this observation data to generatecanonical persistent coordinate data, which may be based on the one ormore observations. In some embodiments, canonical persistent coordinatedata may be more accurate and/or reliable than a single observation ofpersistent coordinate data. In some embodiments, canonical persistentcoordinate data may be transmitted to one or more MR systems. Forexample, an MR system may use image recognition and/or location data torecognize that it is located in a room that has corresponding canonicalpersistent coordinate data (e.g., because other MR systems havepreviously mapped the room). In some embodiments, the MR system mayreceive canonical persistent coordinate data corresponding to itslocation from a remote server.

With respect to FIGS. 1A and 1B, environment/world coordinate system 108defines a shared coordinate space for both real environment 100 andvirtual environment 130. In the example shown, the coordinate space hasits origin at point 106. Further, the coordinate space is defined by thesame three orthogonal axes (108X, 108Y, 108Z). Accordingly, a firstlocation in real environment 100, and a second, corresponding locationin virtual environment 130, can be described with respect to the samecoordinate space. This simplifies identifying and displayingcorresponding locations in real and virtual environments, because thesame coordinates can be used to identify both locations. However, insome examples, corresponding real and virtual environments need not usea shared coordinate space. For instance, in some examples (not shown), amatrix (which may include a translation matrix and a Quaternion matrixor other rotation matrix), or other suitable representation cancharacterize a transformation between a real environment coordinatespace and a virtual environment coordinate space.

FIG. 1C illustrates an example MRE 150 that simultaneously presentsaspects of real environment 100 and virtual environment 130 to user 110via mixed reality system 112. In the example shown, MRE 150simultaneously presents user 110 with real objects 122A, 124A, 126A, and128A from real environment 100 (e.g., via a transmissive portion of adisplay of mixed reality system 112); and virtual objects 122B, 124B,126B, and 132 from virtual environment 130 (e.g., via an active displayportion of the display of mixed reality system 112). As above, originpoint 106 acts as an origin for a coordinate space corresponding to MRE150, and coordinate system 108 defines an x-axis, y-axis, and z-axis forthe coordinate space.

In the example shown, mixed reality objects include corresponding pairsof real objects and virtual objects (i.e., 122A/122B, 124A/124B,126A/126B) that occupy corresponding locations in coordinate space 108.In some examples, both the real objects and the virtual objects may besimultaneously visible to user 110. This may be desirable in, forexample, instances where the virtual object presents informationdesigned to augment a view of the corresponding real object (such as ina museum application where a virtual object presents the missing piecesof an ancient damaged sculpture). In some examples, the virtual objects(122B, 124B, and/or 126B) may be displayed (e.g., via active pixelatedocclusion using a pixelated occlusion shutter) so as to occlude thecorresponding real objects (122A, 124A, and/or 126A). This may bedesirable in, for example, instances where the virtual object acts as avisual replacement for the corresponding real object (such as in aninteractive storytelling application where an inanimate real objectbecomes a “living” character).

In some examples, real objects (e.g., 122A, 124A, 126A) may beassociated with virtual content or helper data that may not necessarilyconstitute virtual objects. Virtual content or helper data canfacilitate processing or handling of virtual objects in the mixedreality environment. For example, such virtual content could includetwo-dimensional representations of corresponding real objects; customasset types associated with corresponding real objects; or statisticaldata associated with corresponding real objects. This information canenable or facilitate calculations involving a real object withoutincurring unnecessary computational overhead.

In some examples, the presentation described above may also incorporateaudio aspects. For instance, in MRE 150, virtual monster 132 could beassociated with one or more audio signals, such as a footstep soundeffect that is generated as the monster walks around MRE 150. Asdescribed further below, a processor of mixed reality system 112 cancompute an audio signal corresponding to a mixed and processed compositeof all such sounds in MRE 150, and present the audio signal to user 110via one or more speakers included in mixed reality system 112 and/or oneor more external speakers.

Example Mixed Reality System

Example mixed reality system 112 can include a wearable head device(e.g., a wearable augmented reality or mixed reality head device)comprising a display (which may include left and right transmissivedisplays, which may be near-eye displays, and associated components forcoupling light from the displays to the user's eyes); left and rightspeakers (e.g., positioned adjacent to the user's left and right ears,respectively); an inertial measurement unit (IMU)(e.g., mounted to atemple arm of the head device); an orthogonal coil electromagneticreceiver (e.g., mounted to the left temple piece); left and rightcameras (e.g., depth (time-of-flight) cameras) oriented away from theuser; and left and right eye cameras oriented toward the user (e.g., fordetecting the user's eye movements). However, a mixed reality system 112can incorporate any suitable display technology, and any suitablesensors (e.g., optical, infrared, acoustic, LIDAR, EOG, GPS, magnetic).In addition, mixed reality system 112 may incorporate networkingfeatures (e.g., Wi-Fi capability) to communicate with other devices andsystems, including other mixed reality systems. Mixed reality system 112may further include a battery (which may be mounted in an auxiliaryunit, such as a belt pack designed to be worn around a user's waist), aprocessor, and a memory. The wearable head device of mixed realitysystem 112 may include tracking components, such as an IMU or othersuitable sensors, configured to output a set of coordinates of thewearable head device relative to the user's environment. In someexamples, tracking components may provide input to a processorperforming a Simultaneous Localization and Mapping (SLAM) and/or visualodometry algorithm. In some examples, mixed reality system 112 may alsoinclude a handheld controller 300, and/or an auxiliary unit 320, whichmay be a wearable beltpack, as described further below.

FIGS. 2A-2D illustrate components of an example mixed reality system 200(which may correspond to mixed reality system 112) that may be used topresent an MRE (which may correspond to MRE 150), or other virtualenvironment, to a user. FIG. 2A illustrates a perspective view of awearable head device 2102 included in example mixed reality system 200.FIG. 2B illustrates a top view of wearable head device 2102 worn on auser's head 2202. FIG. 2C illustrates a front view of wearable headdevice 2102. FIG. 2D illustrates an edge view of example eyepiece 2110of wearable head device 2102. As shown in FIGS. 2A-2C, the examplewearable head device 2102 includes an example left eyepiece (e.g., aleft transparent waveguide set eyepiece) 2108 and an example righteyepiece (e.g., a right transparent waveguide set eyepiece) 2110. Eacheyepiece 2108 and 2110 can include transmissive elements through which areal environment can be visible, as well as display elements forpresenting a display (e.g., via imagewise modulated light) overlappingthe real environment. In some examples, such display elements caninclude surface diffractive optical elements for controlling the flow ofimagewise modulated light. For instance, the left eyepiece 2108 caninclude a left incoupling grating set 2112, a left orthogonal pupilexpansion (OPE) grating set 2120, and a left exit (output) pupilexpansion (EPE) grating set 2122. Similarly, the right eyepiece 2110 caninclude a right incoupling grating set 2118, a right OPE grating set2114 and a right EPE grating set 2116. Imagewise modulated light can betransferred to a user's eye via the incoupling gratings 2112 and 2118,OPEs 2114 and 2120, and EPE 2116 and 2122. Each incoupling grating set2112, 2118 can be configured to deflect light toward its correspondingOPE grating set 2120, 2114. Each OPE grating set 2120, 2114 can bedesigned to incrementally deflect light down toward its associated EPE2122, 2116, thereby horizontally extending an exit pupil being formed.Each EPE 2122, 2116 can be configured to incrementally redirect at leasta portion of light received from its corresponding OPE grating set 2120,2114 outward to a user eyebox position (not shown) defined behind theeyepieces 2108, 2110, vertically extending the exit pupil that is formedat the eyebox. Alternatively, in lieu of the incoupling grating sets2112 and 2118, OPE grating sets 2114 and 2120, and EPE grating sets 2116and 2122, the eyepieces 2108 and 2110 can include other arrangements ofgratings and/or refractive and reflective features for controlling thecoupling of imagewise modulated light to the user's eyes.

In some examples, wearable head device 2102 can include a left templearm 2130 and a right temple arm 2132, where the left temple arm 2130includes a left speaker 2134 and the right temple arm 2132 includes aright speaker 2136. An orthogonal coil electromagnetic receiver 2138 canbe located in the left temple piece, or in another suitable location inthe wearable head unit 2102. An Inertial Measurement Unit (IMU) 2140 canbe located in the right temple arm 2132, or in another suitable locationin the wearable head device 2102. The wearable head device 2102 can alsoinclude a left depth (e.g., time-of-flight) camera 2142 and a rightdepth camera 2144. The depth cameras 2142, 2144 can be suitably orientedin different directions so as to together cover a wider field of view.

In the example shown in FIGS. 2A-2D, a left source of imagewisemodulated light 2124 can be optically coupled into the left eyepiece2108 through the left incoupling grating set 2112, and a right source ofimagewise modulated light 2126 can be optically coupled into the righteyepiece 2110 through the right incoupling grating set 2118. Sources ofimagewise modulated light 2124, 2126 can include, for example, opticalfiber scanners; projectors including electronic light modulators such asDigital Light Processing (DLP) chips or Liquid Crystal on Silicon (LCoS)modulators; or emissive displays, such as micro Light Emitting Diode(μLED) or micro Organic Light Emitting Diode (μOLED) panels coupled intothe incoupling grating sets 2112, 2118 using one or more lenses perside. The input coupling grating sets 2112, 2118 can deflect light fromthe sources of imagewise modulated light 2124, 2126 to angles above thecritical angle for Total Internal Reflection (TIR) for the eyepieces2108, 2110. The OPE grating sets 2114, 2120 incrementally deflect lightpropagating by TIR down toward the EPE grating sets 2116, 2122. The EPEgrating sets 2116, 2122 incrementally couple light toward the user'sface, including the pupils of the user's eyes.

In some examples, as shown in FIG. 2D, each of the left eyepiece 2108and the right eyepiece 2110 includes a plurality of waveguides 2402. Forexample, each eyepiece 2108, 2110 can include multiple individualwaveguides, each dedicated to a respective color channel (e.g., red,blue and green). In some examples, each eyepiece 2108, 2110 can includemultiple sets of such waveguides, with each set configured to impartdifferent wavefront curvature to emitted light. The wavefront curvaturemay be convex with respect to the user's eyes, for example to present avirtual object positioned a distance in front of the user (e.g., by adistance corresponding to the reciprocal of wavefront curvature). Insome examples, EPE grating sets 2116, 2122 can include curved gratinggrooves to effect convex wavefront curvature by altering the Poyntingvector of exiting light across each EPE.

In some examples, to create a perception that displayed content isthree-dimensional, stereoscopically-adjusted left and right eye imagerycan be presented to the user through the imagewise light modulators2124, 2126 and the eyepieces 2108, 2110. The perceived realism of apresentation of a three-dimensional virtual object can be enhanced byselecting waveguides (and thus corresponding the wavefront curvatures)such that the virtual object is displayed at a distance approximating adistance indicated by the stereoscopic left and right images. Thistechnique may also reduce motion sickness experienced by some users,which may be caused by differences between the depth perception cuesprovided by stereoscopic left and right eye imagery, and the autonomicaccommodation (e.g., object distance-dependent focus) of the human eye.

FIG. 2D illustrates an edge-facing view from the top of the righteyepiece 2110 of example wearable head device 2102. As shown in FIG. 2D,the plurality of waveguides 2402 can include a first subset of threewaveguides 2404 and a second subset of three waveguides 2406. The twosubsets of waveguides 2404, 2406 can be differentiated by different EPEgratings featuring different grating line curvatures to impart differentwavefront curvatures to exiting light. Within each of the subsets ofwaveguides 2404, 2406 each waveguide can be used to couple a differentspectral channel (e.g., one of red, green and blue spectral channels) tothe user's right eye 2206. (Although not shown in FIG. 2D, the structureof the left eyepiece 2108 is analogous to the structure of the righteyepiece 2110.)

FIG. 3A illustrates an example handheld controller component 300 of amixed reality system 200. In some examples, handheld controller 300includes a grip portion 346 and one or more buttons 350 disposed along atop surface 348. In some examples, buttons 350 may be configured for useas an optical tracking target, e.g., for tracking six-degree-of-freedom(6DOF) motion of the handheld controller 300, in conjunction with acamera or other optical sensor (which may be mounted in a head unit(e.g., wearable head device 2102) of mixed reality system 200). In someexamples, handheld controller 300 includes tracking components (e.g., anIMU or other suitable sensors) for detecting position or orientation,such as position or orientation relative to wearable head device 2102.In some examples, such tracking components may be positioned in a handleof handheld controller 300, and/or may be mechanically coupled to thehandheld controller. Handheld controller 300 can be configured toprovide one or more output signals corresponding to one or more of apressed state of the buttons; or a position, orientation, and/or motionof the handheld controller 300 (e.g., via an IMU). Such output signalsmay be used as input to a processor of mixed reality system 200. Suchinput may correspond to a position, orientation, and/or movement of thehandheld controller (and, by extension, to a position, orientation,and/or movement of a hand of a user holding the controller). Such inputmay also correspond to a user pressing buttons 350.

FIG. 3B illustrates an example auxiliary unit 320 of a mixed realitysystem 200. The auxiliary unit 320 can include a battery to provideenergy to operate the system 200, and can include a processor forexecuting programs to operate the system 200. As shown, the exampleauxiliary unit 320 includes a clip 2128, such as for attaching theauxiliary unit 320 to a user's belt. Other form factors are suitable forauxiliary unit 320 and will be apparent, including form factors that donot involve mounting the unit to a user's belt. In some examples,auxiliary unit 320 is coupled to the wearable head device 2102 through amulticonduit cable that can include, for example, electrical wires andfiber optics. Wireless connections between the auxiliary unit 320 andthe wearable head device 2102 can also be used.

In some examples, mixed reality system 200 can include one or moremicrophones to detect sound and provide corresponding signals to themixed reality system. In some examples, a microphone may be attached to,or integrated with, wearable head device 2102, and may be configured todetect a user's voice. In some examples, a microphone may be attachedto, or integrated with, handheld controller 300 and/or auxiliary unit320. Such a microphone may be configured to detect environmental sounds,ambient noise, voices of a user or a third party, or other sounds.

FIG. 4 shows an example functional block diagram that may correspond toan example mixed reality system, such as mixed reality system 200described above (which may correspond to mixed reality system 112 withrespect to FIG. 1 ). As shown in FIG. 4 , example handheld controller400B (which may correspond to handheld controller 300 (a “totem”))includes a totem-to-wearable head device six degree of freedom (6DOF)totem subsystem 404A and example wearable head device 400A (which maycorrespond to wearable head device 2102) includes a totem-to-wearablehead device 6DOF subsystem 404B. In the example, the 6DOF totemsubsystem 404A and the 6DOF subsystem 404B cooperate to determine sixcoordinates (e.g., offsets in three translation directions and rotationalong three axes) of the handheld controller 400B relative to thewearable head device 400A. The six degrees of freedom may be expressedrelative to a coordinate system of the wearable head device 400A. Thethree translation offsets may be expressed as X, Y, and Z offsets insuch a coordinate system, as a translation matrix, or as some otherrepresentation. The rotation degrees of freedom may be expressed assequence of yaw, pitch and roll rotations, as a rotation matrix, as aquaternion, or as some other representation. In some examples, thewearable head device 400A; one or more depth cameras 444 (and/or one ormore non-depth cameras) included in the wearable head device 400A;and/or one or more optical targets (e.g., buttons 350 of handheldcontroller 400B as described above, or dedicated optical targetsincluded in the handheld controller 400B) can be used for 6DOF tracking.In some examples, the handheld controller 400B can include a camera, asdescribed above; and the wearable head device 400A can include anoptical target for optical tracking in conjunction with the camera. Insome examples, the wearable head device 400A and the handheld controller400B each include a set of three orthogonally oriented solenoids whichare used to wirelessly send and receive three distinguishable signals.By measuring the relative magnitude of the three distinguishable signalsreceived in each of the coils used for receiving, the 6DOF of thewearable head device 400A relative to the handheld controller 400B maybe determined. Additionally, 6DOF totem subsystem 404A can include anInertial Measurement Unit (IMU) that is useful to provide improvedaccuracy and/or more timely information on rapid movements of thehandheld controller 400B.

In some examples, it may become necessary to transform coordinates froma local coordinate space (e.g., a coordinate space fixed relative to thewearable head device 400A) to an inertial coordinate space (e.g., acoordinate space fixed relative to the real environment), for example inorder to compensate for the movement of the wearable head device 400Arelative to the coordinate system 108. For instance, suchtransformations may be necessary for a display of the wearable headdevice 400A to present a virtual object at an expected position andorientation relative to the real environment (e.g., a virtual personsitting in a real chair, facing forward, regardless of the wearable headdevice's position and orientation), rather than at a fixed position andorientation on the display (e.g., at the same position in the rightlower corner of the display), to preserve the illusion that the virtualobject exists in the real environment (and does not, for example, appearpositioned unnaturally in the real environment as the wearable headdevice 400A shifts and rotates). In some examples, a compensatorytransformation between coordinate spaces can be determined by processingimagery from the depth cameras 444 using a SLAM and/or visual odometryprocedure in order to determine the transformation of the wearable headdevice 400A relative to the coordinate system 108. In the example shownin FIG. 4 , the depth cameras 444 are coupled to a SLAM/visual odometryblock 406 and can provide imagery to block 406. The SLAM/visual odometryblock 406 implementation can include a processor configured to processthis imagery and determine a position and orientation of the user'shead, which can then be used to identify a transformation between a headcoordinate space and another coordinate space (e.g., an inertialcoordinate space). Similarly, in some examples, an additional source ofinformation on the user's head pose and location is obtained from an IMU409. Information from the IMU 409 can be integrated with informationfrom the SLAM/visual odometry block 406 to provide improved accuracyand/or more timely information on rapid adjustments of the user's headpose and position.

In some examples, the depth cameras 444 can supply 3D imagery to a handgesture tracker 411, which may be implemented in a processor of thewearable head device 400A. The hand gesture tracker 411 can identify auser's hand gestures, for example by matching 3D imagery received fromthe depth cameras 444 to stored patterns representing hand gestures.Other suitable techniques of identifying a user's hand gestures will beapparent.

In some examples, one or more processors 416 may be configured toreceive data from the wearable head device's 6DOF headgear subsystem404B, the IMU 409, the SLAM/visual odometry block 406, depth cameras444, and/or the hand gesture tracker 411. The processor 416 can alsosend and receive control signals from the 6DOF totem system 404A. Theprocessor 416 may be coupled to the 6DOF totem system 404A wirelessly,such as in examples where the handheld controller 400B is untethered.Processor 416 may further communicate with additional components, suchas an audio-visual content memory 418, a Graphical Processing Unit (GPU)420, and/or a Digital Signal Processor (DSP) audio spatializer 422. TheDSP audio spatializer 422 may be coupled to a Head Related TransferFunction (HRTF) memory 425. The GPU 420 can include a left channeloutput coupled to the left source of imagewise modulated light 424 and aright channel output coupled to the right source of imagewise modulatedlight 426. GPU 420 can output stereoscopic image data to the sources ofimagewise modulated light 424, 426, for example as described above withrespect to FIGS. 2A-2D. The DSP audio spatializer 422 can output audioto a left speaker 412 and/or a right speaker 414. The DSP audiospatializer 422 can receive input from processor 419 indicating adirection vector from a user to a virtual sound source (which may bemoved by the user, e.g., via the handheld controller 320). Based on thedirection vector, the DSP audio spatializer 422 can determine acorresponding HRTF (e.g., by accessing a HRTF, or by interpolatingmultiple HRTFs). The DSP audio spatializer 422 can then apply thedetermined HRTF to an audio signal, such as an audio signalcorresponding to a virtual sound generated by a virtual object. This canenhance the believability and realism of the virtual sound, byincorporating the relative position and orientation of the user relativeto the virtual sound in the mixed reality environment—that is, bypresenting a virtual sound that matches a user's expectations of whatthat virtual sound would sound like if it were a real sound in a realenvironment.

In some examples, such as shown in FIG. 4 , one or more of processor416, GPU 420, DSP audio spatializer 422, HRTF memory 425, andaudio/visual content memory 418 may be included in an auxiliary unit400C (which may correspond to auxiliary unit 320 described above). Theauxiliary unit 400C may include a battery 427 to power its componentsand/or to supply power to the wearable head device 400A or handheldcontroller 400B. Including such components in an auxiliary unit, whichcan be mounted to a user's waist, can limit the size and weight of thewearable head device 400A, which can in turn reduce fatigue of a user'shead and neck.

While FIG. 4 presents elements corresponding to various components of anexample mixed reality system, various other suitable arrangements ofthese components will become apparent to those skilled in the art. Forexample, elements presented in FIG. 4 as being associated with auxiliaryunit 400C could instead be associated with the wearable head device 400Aor handheld controller 400B. Furthermore, some mixed reality systems mayforgo entirely a handheld controller 400B or auxiliary unit 400C. Suchchanges and modifications are to be understood as being included withinthe scope of the disclosed examples.

Session Manager

MR systems may be uniquely positioned to enable interactive virtualcollaboration between users. Because MR systems may present virtualcontent three-dimensionally and in a user's physical environment, MRcollaboration systems and methods may enable remote collaboration thatcan be at least as effective as local collaboration. In someembodiments, MR collaboration can allow users to see and/or manipulatevirtual content in three-dimensional space. For example, a first usermay launch an MR collaboration session and may see two virtual 3Dmodels, a text document, and a messaging interface. A second user mayjoin the session locally (e.g., the second user may walk into the sameroom as the first user), and the second user may see the same twovirtual 3D models, text document, and messaging interface in the samelocation as the first user. In some embodiments, a third user may jointhe session remotely (e.g., the third user may not be in the same roomas the first and the second users), and the third user may see the twovirtual 3D models, text document, and messaging interface in the thirduser's environment. In some embodiments, the virtual content may sharespatial relationships with each other (e.g., the virtual content may bearranged the same way) for all session users. In some embodiments, MRcollaboration may allow users in the same physical space to leverage theshared physical context to enjoy more meaningful shared experiencesinvolving virtual content.

In some embodiments, displaying and/or synchronizing virtual contentacross multiple MR systems may pose challenges. For example, it can bebeneficial to develop systems and methods for ensuring each MR systemdisplays shared virtual content in a manner that is consistent withother MR systems in a session. It can also be beneficial to developsystems and methods that may enable cross-application collaboration(e.g., virtual content that may be generated using applications createdby different developers). In some embodiments, it can be beneficial todevelop systems and methods that may allow users that are local to eachother (e.g., users that are in the same room) to collaborate with eachother as well as with users that are remote (e.g., in a different room).In some embodiments, it can be beneficial to develop systems and methodsthat may enable collaboration sessions to persist over time such thatsession users may continue collaborating at a later time. In someembodiments, it can be beneficial to develop systems and methods thatmay enable content persistence such that a session user to continueworking on virtual content even without collaborating live with otherusers.

In some embodiments, a session may be broadly defined as a group ofusers (with identifiers) that can collaborate and share a series ofexperiences over time and space. In some embodiments, a session caninclude a communication and collaboration experience that providesnetwork connectivity, common spatial references and a centralized userinterface for chatting and sharing prisms with other MR users. Sessionparticipants can be remote or local in the same physical location. Insome embodiments, a session manager can include a centralized backendservice that manages some or all activity within a session. In someembodiments, session manager can include one or more user-facing,front-end controls and/or expressions representing session managerand/or configured to receive user input (e.g., a menu and/or a sessionhandle). In some embodiments, session manager can include a backgroundservice and/or daemon that orchestrates and manages various sessionevents through various session states. Session manager may also drivethe user experience by allowing users to be discovered and get connectedwith other users. In some embodiments, session manager may also managevarious UI components such as a menu and/or session UI related states.

In some embodiments, collaboration can be facilitated by configuringvirtual content in a collaboration session to behave similarly to realobjects in collaboration sessions. For example, in a “real”collaboration session, users may sit around a table with documentsand/or objects. Users may refer to “this” document and/or “that”document by pointing at a particular document. In some embodiments,users in a real collaboration session may refer to objects usingrelational terms (e.g., that object to the right). This behavior mayoccur naturally to users as a result of years of conditioning andworking physically with other people. It can therefore be desirable todevelop systems and methods for MR collaboration to enable naturalinteractions between users and the content on which they arecollaborating on. In some embodiments, MR collaboration sessions canenable users to refer to collocated virtual content (e.g., virtualcontent that may appear in the same position in a real environment tomultiple users) as if it were real content present in the user'sphysical environment. In some embodiments, MR collaboration sessions canpersist. For example, all users may exit a session, and a user maylaunch the same session several weeks later. In some embodiments, theuser may see all virtual content in the state at which it existed (e.g.,in the same relative positions and/or with the same edits) when theusers previously exited the session.

In some embodiments, a session can include a platform for presenting,synchronizing, managing, and/or storing virtual content used in a mixedreality collaboration session. For example, session users may have arecurring weekly meeting in which virtual content (e.g., word documents,3D models, presentation slides, conversation history, etc.) arediscussed and/or worked on. In some embodiments, users may leverage theplatform of sessions to consolidate virtual content (which may becreated by different developers) into a single virtual space that maypersist over time. For example, loading a single session instance maypresent to a user a 3D model (generated using a first applicationcreated by a first developer), a text document describing goals and/orchanges to the 3D model (generated using a second application created bya second developer), and a conversation history between session usersrelated to this session. This virtual content may persist across timeand across session users, such that the same user or a different sessionuser may load the session and see the same session contents as any othersession user. In some embodiments, a session may enable user presenceflexibility (e.g., local users may share virtual content placement intheir local space, but remote users may also see virtual content withthe same spatial relationships in their remote space). In someembodiments, a session may enable capability flexibility. For example,capabilities (e.g., corresponding to third-party applications) can beinteracted with/enabled/disabled without leaving a centralized sessionplatform. In some embodiments, applications (e.g., third-partyapplications) may leverage the session platform to forgo buildingproprietary sharing platforms that may not be compatible with otherapps. In some embodiments, a session may enable temporal flexibility.For example, users may access sessions at different times, and a livecall with other users may not be necessary. In some embodiments, changesmade by users can be synchronized such that the change may be reflectedfor other session users (whether they are currently in the session orenter the session at a later time).

In some embodiments, a session may include virtual content shared withone or more users over time. A session may have one or more owners, andin some embodiments, a user who created the session may be considered asession owner. A session may have one or more participants who may haveaccess to the session. In some embodiments, a session owner may controlwhat participants may join the session. In some embodiments, a sessionmay have a session identifier. In some embodiments, each user (e.g.,owner or participant) may have a user identifier. In some embodiments, asession may include one or more user avatars, which may represent aremote user's positioning relative to other objects in a session. Insome embodiments, a session may include location data (e.g., locationdata corresponding to each user, location data corresponding tolocations the session has been opened in, etc.). Location data mayinclude persistent coordinate data. In some embodiments, location datamay include one or more transforms (e.g., one or more transformationmatrices), which may relate a position to persistent coordinate data.

In some embodiments, a session can include one or more capabilities. Asession capability may include one or more features that users canselect and/or enable in a session. For example, virtual object sharingmay be considered a session capability. In some embodiments, determiningwhether users are local to other users may be considered a sessioncapability. In some embodiments, projecting a user avatar may beconsidered a session capability. In some embodiments, casting a user'sscreen to other users may be considered a session capability. In someembodiments, a capability can have one or more capability instances(e.g., a capability can have multiple instances running at the sametime). For example, two virtual objects may be shared with users in asession, and each virtual object may be considered a separate capabilityinstance.

In some embodiments, a session may be persistent. For example, a sessionmay continue to exist even after all users have exited a session. Insome embodiments, a session may continue to store session informationsuch as session capabilities used (e.g., sharing a virtual object, whatposition the virtual object was in, etc.), user locations, useridentifications, etc. Persistent sessions may facilitate long-termcollaboration between users. For example, users may continue where theyleft off without having to rearrange their virtual workspace to theirpreference. In some embodiments, session persistence may enable adifferent user to enter the session at a later time and see virtualcontent arranged as it was when a previous user exited the session.

FIGS. 5A-5C illustrate an exemplary MR collaboration session, accordingto some embodiments. FIG. 5A illustrates an exemplary mixed realitycollaboration session where users 508 a, 508 b, and 508 c may be at afirst location (e.g., a first room) together. FIG. 5B illustrates anexemplary mixed reality collaboration session where users 508 d and 508e may be at a second location (e.g., a second room) together. FIG. 5Cillustrates an exemplary mixed reality collaboration session where asession handle has been moved.

In some embodiments, users 508 a, 508 b, 508 c, 508 d, and 508 e may allbe part of the same mixed reality collaboration session 500. In someembodiments, a collaboration session can include a session handle 502 a(which may be a virtual object). Session handle 502 a may serve as alocal anchor for a session. For example, all session users in the samelocation (e.g., users 508 a, 508 b, and 508 c may be considered in thesame location if they share common persistent coordinate data) may bepresented virtual content positioned relative to session handle 502 a,which may give the virtual content the appearance of being located in aparticular location and orientation in the real world, similar to areal/physical object. In some embodiments, session handle 502 a may bepositioned relative to persistent coordinate data (e.g., using atransform). In some embodiments, users 508 a, 508 b, and 508 c may beusing canonical persistent coordinate data, which may enable consistentplacement of session handle 502 a in each user's MR system. In someembodiments, users 508 a, 508 b, and 508 c may all see session handle502 a at the same location (e.g., the users may all see session handle502 a on the floor at the same location).

In some embodiments, whether users can be considered local to each othermay be determined using persistent coordinate data. For example, an MRsystem for user 508 a may receive (e.g., from one or more remoteservers) canonical persistent coordinate data based on an identifiedenvironment for user 508 a. An MR system for user 508 a may use locationdata (e.g., GPS, WiFi, and/or cellular data) and/or image recognitiondata (e.g., recognizing a known environment by comparing captured imageswith images of known environments) to identify an environment for user508 a. In some embodiments, an MR system for user 508 a may transmit itsreceived persistent coordinate data to other MR systems in a session(e.g., an MR system for user 508 b). In some embodiments, other MRsystems in a session may receive canonical persistent coordinate dataand compare the transmitted data received from other MR systems withcanonical persistent coordinate already in use (and/or canonicalpersistent coordinate data received from one or more remote servers). Ifit is determined (e.g., using unique identifiers) that one or moreinstances of canonical persistent coordinate data is shared between MRsystems in a session, it can be determined that the MR systems are localto each other. In some embodiments, if MR systems do not share instancesof canonical persistent coordinate data, it may be determined that theMR systems are remote from each other. In some embodiments, a sessionhandle (e.g., session handle 502 a) may be displayed in relation to oneor more shared instances of persistent canonical persistent coordinatedata, which may enable session handle 502 a to be presented in the samelocation to users 508 a, 508 b, and 508 c.

In some embodiments, session 500 can include a shared virtual object 504a. Shared virtual object 504 a may be considered a session capabilityinstance. In some embodiments, users 508 a, 508 b, and 508 c may all seevirtual object 504 a in the same location (e.g., the users may all seevirtual object 504 a at the end of a real table). In some embodiments,shared virtual object 504 a may be positioned relative to session handle502 a (e.g., using a transform). In some embodiments, shared virtualobject 504 a may be positioned relative to persistent coordinate data(e.g., canonical persistent coordinate data). In some embodiments, auser (e.g., user 508 c) may manipulate shared virtual object 504 a. Forexample, user 508 c may move object 504 a from the edge of the table tothe center of the table. In some embodiments, users 508 a and 508 b mayalso see object 504 a move from the edge of the table to the center ofthe table. In some embodiments, if a user (e.g., user 508 b) points to aportion of object 504 a (e.g., the helmet), other users (e.g., 508 a and508 c) may also see user 508 b as pointing at the same portion of object504 a.

In some embodiments, session handle 502 a may also be moved. Forexample, in FIG. 5C, user 508 a may move session handle 502 a to theleft. In some embodiments, any virtual content displayed as part of asession may also move, thereby maintaining the same relative positioningto session handle 502 a. For example, as session handle 502 a is movedto the left, object 504 a may also be moved to the left by the sameamount. In some embodiments, moving a session handle at one location(e.g., session handle 502 a) may not move a session handle at adifferent location (e.g., session handle 502 b). It can be beneficial toallow each group of local users to manage their own session handleplacement. For example, because virtual content may be positionedrelative to a session handle, each local group may determine an optimallocation for their virtual content for their respective local physicalenvironments.

Session 500 can involve users that may not share the same location. Forexample, in FIG. 5B, users 508 d and 508 e may also be part of session500. In some embodiments, users 508 d and 508 e may be considered remoteto users 508 a, 508 b, and 508 c (e.g., because there may not be commonpersistent coordinate data between users 508 d/508 e and 508 a/508 b/508c). In some embodiments, users 508 d and 508 e may see a second sessionhandle 502 b. In some embodiments, each user (or group of users) thatdoes not have common persistent coordinate data with other users (orgroups of users) may see their own session handle. Shared virtualcontent displayed to users 508 d and 508 e may be displayed relative tosession handle 502 b. For example, shared virtual object 504 b maycorrespond to object 504 a. In some embodiments, object 504 b may bepositioned in the same spot relative to session handle 502 b as object504 a is positioned relative to session handle 502 a. In someembodiments, if object 504 a is moved relative to session handle 502 a,object 504 b may also move relative to session handle 502 b (and viceversa). In some embodiments, session handle 502 b may not move ifsession handle 502 a is moved. This may enable local users to manage howsession contents are presented to the local group of users.

In some embodiments, session 500 can include a user avatar 506 e. Insome embodiments, user avatar 506 e can represent a user in session 500that may be remote to other users in the session. For example, users 508a, 508 b, and 508 c may be considered local to each other (e.g., becausethey may share persistent coordinate data), and user 508 e may beconsidered remote from users 508 a, 508 b, and 508 c (e.g., because user508 e may not share persistent coordinate data with the other users). Insome embodiments, user 508 e (in FIG. 5B) may also be part of session500, and user avatar 506 e may correspond to user 508 e.

In some embodiments, user avatar 506 e may enable user 508 e tocollaborate with users 508 a, 508 b, and 508 c. In some embodiments,avatar 506 e may mirror one or more movement of user 508 e. For example,as user 508 e approaches session handle 502 b, user avatar 506 e mayapproach session handle 502 a, thereby maintaining the same relativepositioning between user 508 e and session handle 502 b. In someembodiments, user 508 e may point to object 504 b, and avatar 506 e maycorrespondingly point to object 504 a at the same location. Similarly,avatar 506 b may represent user 508 b, and avatar 506 a may representuser 508 a. As user 508 a approaches object 504 a, avatar 506 a may alsoapproach object 504 b accordingly. In some embodiments, a remote usermay not broadcast an avatar to other users. For example, user 508 d maybe remote to users 508 a, 508 b, and 508 c, but user 508 d may notproject a corresponding avatar for session handle 502 a.

In some embodiments, session persistence may allow users to dynamicallylocalize to different session locations. For example, users 508 a, 508b, and 508 c may be in a first room, and users 508 d and 508 e may be ina second room, which may be down the hall from the first room. In someembodiments, user 508 a may leave the first room, walk down the hall andenter the second room, and virtual content may be displayed to user 508a relative to session handle 502 b. In some embodiments, each MR systemused by a user may periodically poll the user's location (e.g., usingGPS data and/or image recognition). In some embodiments, an MR systemmay trigger a new location query (e.g., by using geofencing).

FIG. 6 illustrates an exemplary session manager architecture, accordingto some embodiments. In some embodiments, session manager 604 may run onMR system 602, which may include one or more computer systems and cancorrespond to MR systems 112, 200. In some embodiments, session manager604 can include a process, sub-process, thread, and/or service. In someembodiments, session manager 604 can include one or more data structuresconfigured to store information. In some embodiments, session manager604 can include a service (e.g., a background operating system service).In some embodiments, a process, sub-process, thread, and/or service ofsession manager 604 can be configured to continually run (e.g., in thebackground) while an operating system of a host system is running. Insome embodiments, session manager 604 can include an instantiation of aparent background service, which may serve as a host process to one ormore background processes and/or sub-processes. In some embodiments,session manager 604 can include a sub-process of a parent process. Insome embodiments, session manager 604 can include a thread of a parentprocess.

Session manager 604 may include one or more session instances 606 aand/or 606 b. In some embodiments, a session instance can correspond toan MR collaboration session (e.g., session 500). In some embodiments, asession instance may manage information used in an MR collaborationsession. In some embodiments, a session instance may include one or moredata structures configured to store information. In some embodiments, asession instance may include one or more processes, sub-processes,threads, and/or services. In some embodiments, one or more sessioninstances may be stored at one or more remote servers. In someembodiments, session instances may be encrypted before it is stored(locally at an MR device or at one or more remote servers).

In some embodiments, a session instance may be configured to communicatewith one or more capability instances. For example, session instance 606b may be configured to communicate with capability instances 608 b and608 c. A capability instance may correspond to one or more sessioncapabilities. For example, capability instance 608 b may correspond toshared object 504 a. In some embodiments, a capability instance mayinclude one or more data structures configured to store information. Insome embodiments, a capability instance may include one or moreprocesses, sub-processes, threads, and/or services.

In some embodiments, a capability instance can be configured tocommunicate with one or more connectivity services, such as applicationconnectivity platform 610 a and/or collaboration core 610 b. In someembodiments, application connectivity platform 610 a and/orcollaboration core 610 b can include a process, sub-process, thread,and/or service. In some embodiments, application connectivity platform610 a and/or collaboration core 610 b can include one or more datastructures configured to store information. In some embodiments,application connectivity platform 610 a and/or collaboration core 610 bcan include a service (e.g., a background operating system service). Insome embodiments, a process, sub-process, thread, and/or service ofapplication connectivity platform 610 a and/or collaboration core 610 bcan be configured to continually run (e.g., in the background) while anoperating system of a host system is running. In some embodiments,application connectivity platform 610 a and/or collaboration core 610 bcan include an instantiation of a parent background service, which mayserve as a host process to one or more background processes and/orsub-processes. In some embodiments, application connectivity platform610 a and/or collaboration core 610 b can include a sub-process of aparent process. In some embodiments, application connectivity platform610 a and/or collaboration core 610 b can include a thread of a parentprocess.

In some embodiments, application connectivity platform 610 a can providea low-latency communication pathway between MR systems in a colocationsession to enable real-time virtual object colocation. In someembodiments, application connectivity platform 610 a can include one ormore implementations of Web Real-Time Communication (“WebRTC”). Forexample, in some embodiments, data may be transmitted via one or moreTwilio tracks for low-latency communication. In some embodiments,capability instances may utilize application connectivity platform 610 ato send and/or receive low-latency data (e.g., relational transform dataas a shared virtual object moves) from MR systems in a session. In someembodiments, application connectivity platform 610 a can be configuredto communicate with other application connectivity platforms running onother MR systems.

In some embodiments, collaboration core 610 b can provide datasynchronization services for simultaneous edits. In some embodiments,collaboration core 610 b can be configured to receive edit data from oneor more capability instances. In some embodiments, collaboration core610 b can be configured to communicate with external synchronizationservices (e.g., Firebase) to synchronize simultaneous edits to virtualcontent in a session.

In some embodiments, application connectivity platform 610 a and/orcollaboration core 610 b may communicate with session manager 604. Insome embodiments, session manager 604 may provide privileged informationdirectly to application connectivity platform 610 a and/or collaborationcore 610 b (e.g., user identification data). It can be beneficial toshield privileged information from capability instances because acapability instance may be developed by an unknown developer, which maypose a security risk to the privileged data.

Although application connectivity platform 610 a and collaboration core610 b are depicted as separate services, it is also contemplated thatfunctions provided by each could be provided as a single service or morethan two services.

In some embodiments, session manager 604 may communicate with one ormore remote servers and/or with one or more MR systems to synchronizesession instances. For example, a second MR system may initiate asession and invite MR system 602 to participate in the session. In someembodiments, session manager 604 may create a new session instancecorresponding to the newly joined session. In some embodiments, the newsession instance may be a copy of a session instance on the second MRsystem. In some embodiments, a session instance may be received from oneor more remote servers. In some embodiments, session instance data maybe transmitted to one or more remote servers (e.g., if a capabilityinstance has been updated, it can be desirable to transmit the update toother session users). In some embodiments, session instance data can betransmitted to one or more remote servers at an end of a session (e.g.,when the last user leaves a session), so that session data may bepreserved and re-accessed at a later time. In some embodiments, sessionmanager and/or a session instance may communicate with one or moreservices (e.g., one or more services provided by applicationconnectivity platform 610 a) to synchronize session instance data withother session instances (that may be stored at another MR system or aremote server). In some embodiments, session manager and/or a sessioninstance may communicate with one or more services to establish areal-time and/or low-latency communication link with one or more remoteend points (e.g., other MR systems in a session).

FIG. 7 illustrates an exemplary session instance architecture, accordingto some embodiments. In some embodiments, session instance 702 maycorrespond to session instance 604 a and/or 604 b. In some embodiments,session instance 702 can include one or more data structures, which canbe configured to store one or more additional data structures (e.g.,capabilities module 704, participants module 708, locations module 712,and/or presence module 716). Capabilities module 704 may manage dataand/or data structures corresponding to one or more capability instancesin a session. For example, instance 706 a may correspond to a virtualobject. In some embodiments, instance 706 a may include transform data,which may relate the virtual object's position to persistent coordinatedata and/or one or more session handle locations. In some embodiments,instance 706 a may include one or more references to a collaborationcore service. In some embodiments, references to a collaboration coreservice may enable instance 706 a to be properly notified and/or updatedif a change is made to instance 706 a by a user. In some embodiments,instance 706 a may include application connectivity platform data (e.g.,where data should be sent to, what pipes should be used, etc.). In someembodiments, capabilities module 704 may be configured to communicatewith one or more capability instances (e.g., capability instance 608 a).

In some embodiments, session instance 702 may include participantsmodule 708. Participants module 708 may manage data and/or datastructures corresponding to one or more users in a session. For example,user 710 a may include an identifier for an MR system used by a user. Insome embodiments, user 710 a may include avatar data (e.g., appearance,size, color, etc.). In some embodiments, user 710 a may include locationdata. In some embodiments, location data can include GPS data, WiFidata, cellular data, persistent coordinate data, etc.

In some embodiments, session instance 702 may include locations module712. Locations module 712 may manage data and/or data structurescorresponding to one or more locations in a session. For example,location 714 a may include persistent coordinate data, transformationdata, data corresponding to a floor plane, etc. In some embodiments,location 714 a may correspond to a user location. In some embodiments,location 714 a may correspond to a session handle location.

In some embodiments, session instance 702 may include presence module716. Presence module 716 may manage data and/or data structurescorresponding to local and/or remote status of one or more users. Forexample, instance 718 a may indicate that a first user is remote from asecond user, and a third user is local to the second user. In someembodiments, instance 718 a may include data used for communicationbetween users (e.g., using application connectivity platform 610 a).

3D Object Annotation

MR collaboration can be especially useful for 3D virtual contentcreation. Leveraging virtual object persistence, MR systems may enableusers to see virtual content as if the virtual content was real. Forexample, a virtual object may be displayed as resting on a real table.In some embodiments, a user could walk around the table and observe thevirtual object from different angles as if the virtual object was reallysitting on the table. This ability to naturally view and/or interactwith virtual content may be superior to other methods. For example,viewing a 3D model on a 2D screen can require a number of workarounds.Users may have to use a computer mouse to drag the 3D model around todisplay different viewing angles. However, due to the nature ofdisplaying 3D content on a 2D screen, such an experience can befrustrating as the 3D content may change views in unintended ways. Insome embodiments, MR systems may also enable multiple users tocollaborate on 3D content. For example, two users working on the same 3Dcontent may view the 3D content projected in three-dimensional spaceusing MR systems. In some embodiments, the 3D content may besynchronized and/or positioned the same way for both users of MRsystems. Users may then collaborate by referring to aspects of 3Dcontent, moving around to view different angles, etc. In someembodiments, annotations to virtual content can be made available tocollaborating users in real time. For example, a first user may addvirtual markup and/or comments to virtual content, and a second user maysee the virtual markup and/or comments as the first user creates them.It can therefore be beneficial to develop systems and methods to enablereal-time collaboration on 3D objects.

FIG. 8 illustrates an exemplary mixed reality collaboration session,according to some embodiments. In some embodiments, users 802 and 804may use one or more MR systems (e.g., MR system 806, which cancorrespond to MR systems 112, 200) to collaborate on 3D virtual content.In some embodiments, users 802 and 804 may utilize a session to displayand/or collaborate on virtual content. For example, virtual model 808may be presented to users 802 and 804. In some embodiments, virtualmodel 808 can be presented to users 802 and 804 in the same position(e.g., location and/or orientation). In some embodiments, virtual model808 can be presented to users 802 and 804 using the same session handle.In some embodiments, virtual model 808 may be managed by a capabilityinstance. In some embodiments, properties of virtual model 808 can bestored in and/or managed by a capability instance. In some embodiments,a capability instance can be stored in and/or managed by a sessioninstance.

In some embodiments, user 802 may annotate virtual model 808 (e.g., bycreating virtual markup and/or adding virtual comments). For example,user 802 may create virtual markup 812, which may indicate that a cafécould be placed at a location in virtual model 808. In some embodiments,user 804 may see virtual markup 812. In some embodiments, user 804 maysee virtual markup 812 as user 802 is creating virtual markup 812. Insome embodiments, user 804 may see virtual markup 812 in the samelocation as user 802 sees virtual markup 812. In some embodiments, user802 can create one or more virtual comments. In some embodiments, avirtual comment can include location indicator 814 and/or comment bubble816. In some embodiments, comment bubble 816 can include a visualindicator corresponding to location indicator 814 (e.g., comment bubble816 and location indicator 814 can share a number). In some embodiments,location indicator 814 may be presented to user 804 in the same position(e.g., relative to the real world, and/or relative to other virtualcontent) as it is presented to user 802. In some embodiments, commentbubble 816 can be presented in different positions for different users.For example, comment bubble 816 may be presented to user 802 as facinguser 802, and comment bubble 816 may be presented to user 804 as facinguser 804. In some embodiments, comment bubble 816 may be configured tocontinually face a user as a user looks in different locations. In someembodiments, comment bubble 816 can be presented in the same positionfor multiple users (e.g., all local users) of a session.

In some embodiments, data corresponding to virtual annotations may betransmitted from a capability instance (e.g., capability instance 608 c)to a session instance (e.g., session instance 606 b). In someembodiments, data corresponding to virtual annotations may betransmitted from a capability instance to collaboration core 610 b. Insome embodiments, collaboration core 610 b may transmit datacorresponding to virtual annotations to one or more remote servers(e.g., one or more remote servers configured to handle datasynchronization and/or synchronization conflicts). In some embodiments,one or more remote servers may transmit data corresponding to virtualannotations to other session users. In some embodiments, datacorresponding to virtual annotations can be stored in a sessioninstance. In some embodiments, a session instance can be closed andre-opened, and one or more capability instances (e.g., virtual model 808and/or virtual markup 812) can be loaded and/or displayed to users.

In some embodiments, user 802 may be remote from user 804. For example,user 802 may be in a first room, and user 804 may be in a second roomdifferent than the first room. In some embodiments, users 802 and 804may collaborate on virtual model 808 using a session instance. In someembodiments, user 802 may see virtual model 808 in the first room, anduser 804 may see virtual model 808 in the second room. In someembodiments, virtual model 808 may be presented relative to a firstsession handle for user 802, and virtual model 808 may be presentedrelative to a second session handle for user 804. In some embodiments,virtual annotations made by one user (e.g., user 802) may be visible toall session users (e.g., user 804).

FIG. 9 illustrates an exemplary annotation menu, according to someembodiments. In some embodiments, user 902 may use MR system 904 to viewvirtual object 906. In some embodiments, MR system 904 may displayannotation menu 908, which may allow user 902 to annotate virtual object906. In some embodiments, MR system 904 may dynamically displayannotation menu 908 as facing user 902 regardless of which directionuser 902 is facing. For example, as user 902 moves around object 906,annotation menu 908 may rotate so that the full menu can be visible touser 902. It can be desirable to dynamically orient annotation menu 908because it may be difficult for user 902 to select options on annotationmenu 908 if annotation menu 908 is displayed at an angle to user 902(e.g., because the effective visible area of buttons on annotation menu908 may be too small for user 902 to identify and/or select). In someembodiments, a desired orientation for menu 908 can be determined bydetermining a position (e.g., location and/or orientation) of a head ofuser 902. In some embodiments, menu 908 can be oriented such that anormal vector of menu 908 is directed at user 902. In some embodiments,MR system 904 may display annotation menu 908 in proximity to acorresponding virtual object (e.g., virtual object 906).

In some embodiments, MR system 904 may display annotation menu 908 suchthat annotation menu 908 may not be occluded by virtual object 906. Forexample, if annotation menu 908 is displayed in front of object 906, anduser 902 moves to the opposite side of object 906, annotation menu 908may be wholly or partially occluded by object 906. It can be desirableto dynamically reposition annotation menu 908 such that a correspondingvirtual object (e.g., a virtual object being annotated using menu 908)may not occlude annotation menu 908 (e.g., because user 902 may not beable to interact with menu 908 if it is occluded by a correspondingvirtual object). In some embodiments, occlusion may be determined bydetermining whether virtual object 906 (and/or a prism associated withvirtual object 906) intersects a direct path between user 902 andannotation menu 908.

Annotation menu 908 can have several features for annotating virtualobjects. For example, annotation menu 908 can include a virtual drawbutton. In some embodiments, a virtual draw button can toggle drawingmode, which may enable a user to create virtual markup on and/or arounda virtual object. In some embodiments, virtual markup created in drawmode may be visible to other session users. In some embodiments,annotation menu 908 can include a virtual delete button. In someembodiments, a virtual delete button may delete a selected virtualobject and/or virtual markup. In some embodiments, annotation menu 908can include a virtual comment button. In some embodiments, virtualcomment button may enable a user to place a location indicator for avirtual comment and/or add a virtual comment. In some embodiments,annotation menu 908 can include a virtual color picker button. In someembodiments, selecting a virtual color picker button may enable a userto select a color for virtual markup. In some embodiments, annotationmenu 908 can include a virtual size toggle button. In some embodiments,selecting a virtual size toggle button can toggle whether a virtualobject is displayed life-size. For example, a 3D model may includeparameters such as dimensions. In some embodiments, an MR system maydisplay a 3D model at a modified size to make the 3D model easilyviewable. For example, a 3D model of a building may be initiallypresented to a user as much smaller than its true dimensions so that auser may easily see the entirety of the 3D model. In some embodiments,annotation menu 908 can include a virtual annotation visibility button.In some embodiments, selecting the virtual annotation visibility buttonmay toggle the visibility (e.g., the display may display or not display)of annotations (e.g., virtual markup and/or virtual comments)corresponding to a virtual object. In some embodiments, annotation menu908 can include a virtual clear button. In some embodiments, selectingthe virtual clear button can remove all virtual annotationscorresponding to a virtual object.

In some embodiments, user 902 may move virtual object 906. For example,user 902 may use handheld controller 910 (which can correspond tohandheld controller 300) and a virtual selection beam 912 to selectvirtual object 906 and drag virtual object 906 to a new position. Insome embodiments, annotation menu 908 may move with virtual object 906and maintain a relative position to virtual object 906.

FIG. 10 illustrates an example of mixed reality annotation, according tosome embodiments. In some embodiments, virtual object 1002 may be aprism and may include one or more virtual objects within it (e.g.,virtual object 1004). In some embodiments, a prism may include abounding volume and/or parameters of included virtual objects. In someembodiments, virtual object 1012 can include a prism which may includean annotation menu (which can correspond to annotation menu 908). Insome embodiments, virtual object 1012 may not intersect with prism 1002(e.g., because object 1004 may obscure a view of annotation menu 1012,which may prevent the user from interacting with annotation menu 1012).

In some embodiments, a user may select location indicator 1008, whichmay be displayed on and/or near virtual objects 1002 and/or 1004. Insome embodiments, selecting location indicator 1008 may cause virtualobject 1010 to be displayed to a user. Virtual object 1010 can include aprism, which may include a virtual comment bubble. In some embodiments,virtual object 1010 can be displayed near location indicator 1008. Insome embodiments, virtual object 1010 may not intersect with virtualobject 1002 (e.g., because it may obscure a view of a virtual commentbubble). In some embodiments, virtual objects 1012 and/or 1010 maycontinually face a user as a user moves around an environment. In someembodiments, virtual objects 1012 and/or 1010 may reposition themselvesif their view becomes obstructed (e.g., if a user moves such thatvirtual object 1002 obstructs a view from the user to virtual object1010).

FIG. 11 illustrates an example of mixed reality annotation, according tosome embodiments. In some embodiments, virtual object 1108 (which mayinclude a prism) can be very large (e.g., because an included virtualobject 1102 is very large). In some embodiments, virtual object 1104 maybe displayed within another virtual object (e.g., virtual object 1108).For example, virtual object 1108 may include a bounding prism of virtualobject 1102, but because of the size of virtual object 1102, significantspace may exist where virtual object 1102 does not obstruct vision ofother virtual objects. In some embodiments, virtual object 1104 caninclude a virtual comment bubble, which a user may enter text into. Insome embodiments, virtual keyboard 1106 may be displayed as a user editsa virtual comment bubble. In some embodiments, virtual keyboard 1106 maybe displayed in proximity to a corresponding virtual comment bubble. Forexample, a user may edit comment bubble 1104, and virtual keyboard 1106may be displayed near bubble 1104 (instead of, for example, commentbubble 1106). It can be beneficial to visually indicate which virtualcomment bubble a user is editing (e.g., by displaying a keyboard nearthe corresponding comment bubble).

Example systems, methods, and computer-readable media are disclosed.According to some examples, a system comprises a wearable devicecomprising a transmissive display; one or more processors configured toexecute a method comprising: presenting a virtual object to a first userat a first position via the transmissive display of the wearable device;receiving a first input from the first user; in response to receivingthe first input, presenting a virtual annotation at a first displacementfrom the first position via the transmissive display; transmitting firstdata to a second user, the first data associated with the virtualannotation and the first displacement; receiving a second input from thesecond user; in response to receiving the second input, presenting thevirtual annotation to the first user at a second displacement from thefirst position via the transmissive display; and transmitting seconddata to a remote server, the second data associated with the virtualobject, the virtual annotation, the second displacement, and the firstposition. In some examples, the virtual object is presented based on afirst application configured to run on the wearable device, and thevirtual annotation is presented based on data from a plugin library, theplugin library configured to be accessed by multiple applicationsconfigured to run on the wearable device. In some examples, the seconddata is transmitted at a first time, and the method further comprises:exiting a session instance, wherein the session instance is configuredto store the second data; receiving a third input from the first user;in response to receiving the third input, requesting the second data;presenting the virtual object at the first position to the first user ata second time later than the first time; and presenting the virtualannotation at the second displacement from the first position to thefirst user at the second time. In some examples, the annotationcomprises a virtual markup. In some examples, the virtual object ispresented at a first size, the virtual object comprises target dimensiondata, and the method further comprises: receiving a third input from thefirst user; in response to receiving the third input, presenting thevirtual object at a second size to the first user, wherein the secondsize is associated with the target dimension data; and transmittingthird data to the second user, the third data associated with the secondsize. In some examples, the method further comprises: receiving a thirdinput from the first user; in response to receiving the third input,presenting the virtual location indicator to the first user, wherein thevirtual location indicator is associated with a virtual comment; andtransmitting third data to the second user, the third data associatedwith the virtual location indicator. In some examples, the methodfurther comprises: presenting a virtual annotation menu to the firstuser via the transmissive display; and repositioning the virtualannotation menu such that the virtual annotation menu is not occluded bythe virtual object.

According to some examples, a method comprises presenting a virtualobject to a first user at a first position via a transmissive display ofa wearable device; receiving a first input from the first user; inresponse to receiving the first input, presenting a virtual annotationat a first displacement from the first position via the transmissivedisplay; transmitting first data to a second user, the first dataassociated with the virtual annotation and the first displacement;receiving a second input from the second user; in response to receivingthe second input, presenting the virtual annotation to the first user ata second displacement from the first position via the transmissivedisplay; and transmitting second data to a remote server, the seconddata associated with the virtual object, the virtual annotation, thesecond displacement, and the first position. In some examples, thevirtual object is presented based on a first application configured torun on the wearable device, and the virtual annotation is presentedbased on data from a plugin library, the plugin library configured to beaccessed by multiple applications configured to run on the wearabledevice. In some examples, the second data is transmitted at a firsttime, and the method further comprises: exiting a session instance,wherein the session instance is configured to store the second data;receiving a third input from the first user; in response to receivingthe third input, requesting the second data; presenting the virtualobject at the first position to the first user at a second time laterthan the first time; and presenting the virtual annotation at the seconddisplacement from the first position to the first user at the secondtime. In some examples, the annotation comprises a virtual markup. Insome examples, the virtual object is presented at a first size, thevirtual object comprises target dimension data, and the method furthercomprises: receiving a third input from the first user; in response toreceiving the third input, presenting the virtual object at a secondsize to the first user, wherein the second size is associated with thetarget dimension data; and transmitting third data to the second user,the third data associated with the second size. In some examples, themethod further comprises: receiving a third input from the first user;in response to receiving the third input, presenting the virtuallocation indicator to the first user, wherein the virtual locationindicator is associated with a virtual comment; and transmitting thirddata to the second user, the third data associated with the virtuallocation indicator. In some examples, the method further comprises:presenting a virtual annotation menu to the first user via thetransmissive display; and repositioning the virtual annotation menu suchthat the virtual annotation menu is not occluded by the virtual object.

According to some examples, a non-transitory computer-readable mediumstores instructions that, when executed by one or more processors, causethe one or more processors to execute a method comprising: presenting avirtual object to a first user at a first position via a transmissivedisplay of a wearable device; receiving a first input from the firstuser; in response to receiving the first input, presenting a virtualannotation at a first displacement from the first position via thetransmissive display; transmitting a first data to a second user, thefirst data associated with the virtual annotation and the firstdisplacement; receiving a second input from the second user; in responseto receiving the second input, presenting the virtual annotation to thefirst user at a second displacement from the first position via thetransmissive display; and transmitting second data to a remote server,the second data associated with the virtual object, the virtualannotation, the second displacement, and the first position. In someexamples, the virtual object is presented based on a first applicationconfigured to run on the wearable device, and the virtual annotation ispresented based on data from a plugin library, the plugin libraryconfigured to be accessed by multiple applications configured to run onthe wearable device. In some examples, the second data is transmitted ata first time, and the method further comprises: exiting a sessioninstance, wherein the session instance is configured to store the seconddata; receiving a third input from the first user; in response toreceiving the third input, requesting the second data; presenting thevirtual object at the first position to the first user at a second timelater than the first time; and presenting the virtual annotation at thesecond displacement from the first position to the first user at thesecond time. In some examples, the annotation comprises a virtualmarkup. In some examples, the virtual object is presented at a firstsize, the virtual object comprises target dimension data, and the methodfurther comprises: receiving a third input from the first user; inresponse to receiving the third input, presenting the virtual object ata second size to the first user, wherein the second size is associatedwith the target dimension data; and transmitting third data to thesecond user, the third data associated with the second size. In someexamples, the method further comprises: receiving a third input from thefirst user; in response to receiving the third input, presenting thevirtual location indicator to the first user, wherein the virtuallocation indicator is associated with a virtual comment; andtransmitting third data to the second user, the third data associatedwith the virtual location indicator. In some examples, the methodfurther comprises: presenting a virtual annotation menu to the firstuser via the transmissive display; and repositioning the virtualannotation menu such that the virtual annotation menu is not occluded bythe virtual object.

Although the disclosed examples have been fully described with referenceto the accompanying drawings, it is to be noted that various changes andmodifications will become apparent to those skilled in the art. Forexample, elements of one or more implementations may be combined,deleted, modified, or supplemented to form further implementations. Suchchanges and modifications are to be understood as being included withinthe scope of the disclosed examples as defined by the appended claims.

1. A system comprising: a wearable device comprising a transmissivedisplay; and one or more processors configured to execute a methodcomprising: presenting a virtual object to a user at a first positionvia the transmissive display of the wearable device; at a first time,receiving a first input from the user; in response to receiving thefirst input, presenting a virtual annotation at a first displacementfrom the first position via the transmissive display; transmitting firstdata to a remote server, the first data associated with the virtualobject, the virtual annotation, and the first displacement; at a secondtime later than the first time, receiving a second input from the user;in response to receiving the second input, presenting the virtualannotation to the user at a second displacement from the first positionvia the transmissive display, the second displacement different from thefirst displacement; and transmitting second data to the remote server,the second data associated with the virtual object, the virtualannotation, and the second displacement, wherein: receiving the firstinput from the user comprises receiving the first input via a firstannotation menu dynamically presented to the user at the first time at afixed orientation relative to an orientation of the user at the firsttime, and receiving the second input from the user comprises receivingthe second input via a second annotation menu dynamically presented tothe user at the second time at a fixed orientation relative to anorientation of the user at the second time.
 2. The system of claim 1,wherein presenting the virtual object comprises presenting the virtualobject based on a first application configured to run on the wearabledevice, and wherein presenting the virtual annotation comprisespresenting the virtual annotation based on data from a plugin library,the plugin library configured to be accessed by multiple applicationsconfigured to run on the wearable device.
 3. The system of claim 1,wherein the method further comprises exiting a session instance, thesession instance configured to store the second data.
 4. The system ofclaim 1, wherein the method further comprises, concurrently withpresenting the virtual annotation to the user, presenting a secondvirtual annotation to the user, wherein the second virtual annotationcomprises data received from a second user via the remote server.
 5. Thesystem of claim 1, wherein presenting the virtual object comprisespresenting the virtual object at a first size, wherein the virtualobject comprises target dimension data, and wherein the method furthercomprises: at a third time later than the second time, receiving a thirdinput from the user; in response to receiving the third input,presenting the virtual object at a second size to the user, wherein thesecond size is associated with the target dimension data; andtransmitting third data to the remote server, the third data associatedwith the second size.
 6. The system of claim 1, the method furthercomprising: at a third time later than the second time, receiving athird input from the user; in response to receiving the third input,presenting a virtual location indicator to the user, wherein the virtuallocation indicator is associated with a virtual comment; andtransmitting third data to the remote server, the third data associatedwith the virtual location indicator.
 7. The system of claim 1, whereinthe method further comprises: receiving an indication to present thefirst annotation menu to the user via the transmissive display;determining whether a view of the first annotation menu is occluded bythe virtual object; and in response to determining that the view of thefirst annotation menu is occluded by the virtual object, repositioningthe first annotation menu such that the first annotation menu is notoccluded by the virtual object.
 8. A method comprising: presenting avirtual object to a user at a first position via a transmissive displayof a wearable device; at a first time, receiving a first input from theuser; in response to receiving the first input, presenting a virtualannotation at a first displacement from the first position via thetransmissive display; transmitting first data to a remote server, thefirst data associated with the virtual object, the virtual annotation,and the first displacement; at a second time later than the first time,receiving a second input from the user; in response to receiving thesecond input, presenting the virtual annotation to the user at a seconddisplacement from the first position via the transmissive display, thesecond displacement different from the first displacement; andtransmitting second data to the remote server, the second dataassociated with the virtual object, the virtual annotation, and thesecond displacement, wherein: receiving the first input from the usercomprises receiving the first input via a first annotation menudynamically presented to the user at the first time at a fixedorientation relative to an orientation of the user at the first time,and receiving the second input from the user comprises receiving thesecond input via a second annotation menu dynamically presented to theuser at the second time at a fixed orientation relative to anorientation of the user at the second time.
 9. The method of claim 8,wherein the virtual object is presented based on a first applicationconfigured to run on the wearable device, and wherein the virtualannotation is presented based on data from a plugin library, the pluginlibrary configured to be accessed by multiple applications configured torun on the wearable device.
 10. The method of claim 8, furthercomprising exiting a session instance, the session instance configuredto store the second data.
 11. The method of claim 8, further comprising:concurrently with presenting the virtual annotation to the user,presenting a second virtual annotation to the user, wherein the secondvirtual annotation comprises data received from a second user via theremote server.
 12. The method of claim 8, wherein presenting the virtualobject comprises presenting the virtual object at a first size, whereinthe virtual object comprises target dimension data, and wherein themethod further comprises: at a third time later than the second time,receiving a third input from the user; in response to receiving thethird input, presenting the virtual object at a second size to the user,wherein the second size is associated with the target dimension data;and transmitting third data to the remote server, the third dataassociated with the second size.
 13. The method of claim 8, furthercomprising: at a third time later than the second time, receiving athird input from the user; in response to receiving the third input,presenting a virtual location indicator to the user, wherein the virtuallocation indicator is associated with a virtual comment; andtransmitting third data to the remote server, the third data associatedwith the virtual location indicator.
 14. The method of claim 8, furthercomprising: receiving an indication to present the first annotation menuto the user via the transmissive display; determining whether a view ofthe first annotation menu is occluded by the virtual object; and inresponse to determining that the view of the first annotation menu isoccluded by the virtual object, repositioning the first annotation menusuch that the first annotation menu is not occluded by the virtualobject.
 15. A non-transitory computer-readable medium storinginstructions that, when executed by one or more processors, cause theone or more processors to execute a method comprising: presenting avirtual object to a user at a first position via a transmissive displayof a wearable device; at a first time, receiving a first input from theuser; in response to receiving the first input, presenting a virtualannotation at a first displacement from the first position via thetransmissive display; transmitting first data to a remote server, thefirst data associated with the virtual object, the virtual annotation,and the first displacement; at a second time later than the first time,receiving a second input from the user; in response to receiving thesecond input, presenting the virtual annotation to the user at a seconddisplacement from the first position via the transmissive display, thesecond displacement different from the first displacement; andtransmitting second data to the remote server, the second dataassociated with the virtual object, the virtual annotation, and thesecond displacement, wherein: receiving the first input from the usercomprises receiving the first input via a first annotation menudynamically presented to the user at the first time at a fixedorientation relative to an orientation of the user at the first time,and receiving the second input from the user comprises receiving thesecond input via a second annotation menu dynamically presented to theuser at the second time at a fixed orientation relative to anorientation of the user at the second time.
 16. The non-transitorycomputer-readable medium of claim 15, wherein the virtual object ispresented based on a first application configured to run on the wearabledevice, and wherein the virtual annotation is presented based on datafrom a plugin library, the plugin library configured to be accessed bymultiple applications configured to run on the wearable device.
 17. Thenon-transitory computer-readable medium of claim 15, the method furthercomprising: concurrently with presenting the virtual annotation to theuser, presenting a second virtual annotation to the user, wherein thesecond virtual annotation comprises data received from a second user viathe remote server.
 18. The non-transitory computer-readable medium ofclaim 15, wherein presenting the virtual object comprises presenting thevirtual object at a first size, wherein the virtual object comprisestarget dimension data, and wherein the method further comprises: at athird time later than the second time, receiving a third input from theuser; in response to receiving the third input, presenting the virtualobject at a second size to the user, wherein the second size isassociated with the target dimension data; and transmitting third datato the remote server, the third data associated with the second size.19. The non-transitory computer-readable medium of claim 15, the methodfurther comprising: at a third time later than the second time,receiving a third input from the user; in response to receiving thethird input, presenting a virtual location indicator to the user,wherein the virtual location indicator is associated with a virtualcomment; and transmitting third data to the remote server, the thirddata associated with the virtual location indicator.
 20. Thenon-transitory computer-readable medium of claim 15, the method furthercomprising: receiving an indication to present the first annotation menuto the user via the transmissive display; determining whether a view ofthe first annotation menu is occluded by the virtual object; and inresponse to determining that the view of the first annotation menu isoccluded by the virtual object, repositioning the first annotation menusuch that the first annotation menu is not occluded by the virtualobject.